Information processing device, display method, and nontransitory computer-readable medium for storing program

ABSTRACT

Provided are an information processing device, a display method, and a program which enable easy recognition of the discrepancy between the detection result by detection processing for an image captured by an endoscope and the detection result by a user.An information processing device (1) includes: a first acquisition unit (2) configured to acquire a capturing time of a lesion image instructed to be saved by a user, from a series of images captured by an endoscope during examination with the endoscope; a second acquisition unit (3) configured to acquire a capturing time of a lesion image detected by detection processing for the series of images captured by the endoscope during the examination; and a display control unit (4) configured to cause a display device to display a first capturing time and a second capturing time which are plotted on a time axis, the first capturing time being the capturing time acquired by the first acquisition unit (2), the second capturing time being the capturing time acquired by the second acquisition unit (3).

TECHNICAL FIELD

The present disclosure relates to an information processing device, adisplay method, and a non-transitory computer-readable medium forstoring a program.

BACKGROUND ART

A system is known that supports medical treatment using an endoscope.For example, Patent Literature 1 discloses an image display device thatdisplays an image captured by a capsule-type endoscope. In the imagedisplay device, when an operation of selecting an “Auto” icon isperformed, an image of a bleeding site is automatically extracted.

CITATION LIST Patent Literature

-   Patent Literature 1: Japanese Unexamined Patent Application    Publication No 2006-061469

SUMMARY OF INVENTION Technical Problem

In a case of detection of a lesion by detection processing on an imagecaptured by an endoscope in real time in addition to detection of alesion by a user (for example, a doctor) during in-vivo examinationusing an endoscope, there may be discrepancy between the detectionresult by the user and the detection result by the detection processing.However, it is difficult to easily recognize such discrepancy. On theother hand, in the technique disclosed in Patent Literature 1, only animage of a bleeding site is automatically extracted, and the comparisonbetween the automatic extraction result and the detection result by theuser is not considered.

The present disclosure has been made in order to solve such a problem,and is to provide an information processing device, a display method,and a program which enable easy recognition of the discrepancy betweenthe detection result by detection processing for an image captured by anendoscope and the detection result by a user.

Solution to Problem

An information processing device according to a first aspect of thepresent disclosure includes:

a first acquisition unit configured to acquire a capturing time of alesion image instructed to be saved by a user, from a series of imagescaptured by an endoscope during examination with the endoscope;

a second acquisition unit configured to acquire a capturing time of alesion image detected by detection processing for the series of imagescaptured by the endoscope during the examination; and

a display control unit configured to cause a display device to display afirst capturing time and a second capturing time which are plotted on atime axis, the first capturing time being the capturing time acquired bythe first acquisition unit, the second capturing time being thecapturing time acquired by the second acquisition unit.

A display method according to a second aspect of the present disclosureincludes:

acquiring a capturing time of a lesion image instructed to be saved by auser, from a series of images captured by an endoscope duringexamination with the endoscope;

acquiring a capturing time of a lesion image detected by detectionprocessing for the series of images captured by the endoscope during theexamination; and

causing a display device to display a first capturing time and a secondcapturing time which are plotted on a time axis, the first capturingtime being the capturing time of the lesion image instructed to be savedby the user, the second capturing time being the capturing time of thelesion image detected by the detection processing.

A program according to a third aspect of the present disclosure causes acomputer to execute:

a first acquisition step of acquiring a capturing time of a lesion imageinstructed to be saved by a user, from a series of images captured by anendoscope during examination with the endoscope;

a second acquisition step of acquiring a capturing time of a lesionimage detected by detection processing for the series of images capturedby the endoscope during the examination; and

a display control step of causing a display device to display a firstcapturing time and a second capturing time which are plotted on a timeaxis, the first capturing time being the capturing time acquired in thefirst acquisition step, the second capturing time being the capturingtime acquired in the second acquisition step.

Advantageous Effect of Invention

According to the present disclosure, it is possible to provide aninformation processing device, a display method, and a program whichenable easy recognition of the discrepancy between the detection resultby detection processing for an image captured by an endoscope and thedetection result by a user.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing an example of a configuration of aninformation processing device according to a first example embodiment.

FIG. 2 is a block diagram showing a configuration of an examinationsupport system according to a second example embodiment.

FIG. 3 is a schematic diagram showing an example of a display screengenerated by a processor device.

FIG. 4 is a block diagram showing an example of a functionalconfiguration of an information processing device according to thesecond example embodiment.

FIG. 5 is a schematic diagram showing a display example based on controlof an input/output control unit.

FIG. 6 shows a display example when a scale of a time axis shown in FIG.5 is enlarged.

FIG. 7 shows a display example when a scale of a time axis shown in FIG.6 is enlarged.

FIG. 8 is a schematic diagram showing an example of a hardwareconfiguration of the information processing device according to thefirst example embodiment.

FIG. 9 is a flowchart showing an example of an operation of theinformation processing device according to the first example embodiment.

FIG. 10 is a block diagram showing an example of a functionalconfiguration of an information processing device according to a thirdexample embodiment.

FIG. 11 is a flowchart showing an example of an operation of theinformation processing device according to the third example embodiment.

FIG. 12 is a block diagram showing an example of a functionalconfiguration of an information processing device according to a fourthexample embodiment.

FIG. 13 is a schematic diagram showing a display example when a failuredetection unit detects a failure.

FIG. 14 is a flowchart showing an example of an operation of theinformation processing device according to the fourth exampleembodiment.

FIG. 15 is a block diagram showing an example of a functionalconfiguration of an information processing device according to a fifthexample embodiment.

FIG. 16 is a schematic diagram showing a display example of a screen tobe displayed for generation of feedback data.

FIG. 17 is a schematic diagram showing a display example of a screen tobe displayed for generation of feedback data.

FIG. 18 is a flowchart showing an example of an operation of theinformation processing device according to the fifth example embodiment.

DESCRIPTION OF EMBODIMENTS

For clarifying the explanation, the following descriptions and thedrawings are omitted and simplified as appropriate. In each drawing, thesame or corresponding components are designated by the same referencenumerals, and duplicate descriptions are omitted as necessary for thesake of clarity of explanation. In addition, the features of eachexample embodiment can be combined as long as there is no technicalcontradiction.

First Example Embodiment

FIG. 1 is a block diagram showing an example of a configuration of aninformation processing device 1 according to a first example embodiment.The information processing device 1 is a device that supports anexamination by a user (for example, a doctor) using an endoscope, andperforms display processing related to detection of a lesion. As shownin FIG. 1, the information processing device 1 includes a firstacquisition unit 2, a second acquisition unit 3, and a display controlunit 4.

The first acquisition unit 2 acquires a capturing time of a lesion imagethat the user has instructed to save, from a series of images capturedby an endoscope during examination with the endoscope. The capturingtime may be an absolute time such as a system time at the time ofcapturing, or may be a relative time with reference to a start time ofcapturing. For example, the first acquisition unit 2 may acquire theabove-described capturing time based on log data of the endoscopesystem, or may acquire the capturing time by analyzing a moving imageoutput from the endoscope system to specify the above-describedcapturing time. The lesion image refers to an image including a lesionsite, that is, an image in which the lesion site is detected. Here, thelesion refers to an abnormality in a biological tissue caused by adisease, the abnormality including, for example, a polyp or a tumor, butbeing not limited thereto. The user instructs the endoscope system tosave an image showing the lesion site visually recognized by the userduring the examination with the endoscope. Therefore, the lesion imageinstructed to be saved by the user can be regarded as a lesion imageshowing the lesion site visually recognized by the user.

The second acquisition unit 3 acquires a capturing time of a lesionimage detected by detection processing for a series of images capturedby the endoscope in the midst of examination. The detection processingis processing of performing any image recognition processing to detect alesion site depicted in an image. Therefore, the second acquisition unit3 acquires a capturing time of the image (lesion image) in which thelesion site is detected by the detection processing, in other words, acapturing time of the lesion image in which the lesion site detected bythe detection processing is depicted.

The display control unit 4 controls a display of information on adisplay device. Specifically, the display control unit 4 causes thedisplay device to display a first capturing time which is the capturingtime acquired by the first acquisition unit 2 and a second capturingtime which is the capturing time acquired by the second acquisition unit3 on a time axis in a plotting manner.

According to the present example embodiment, the capturing time of thelesion image that the user has instructed to save during the examinationwith the endoscope, that is, the capturing time of the lesion image forthe lesion detected by the user and the capturing time of the lesionimage for the lesion detected by the detection processing are plotted onthe time axis. Therefore, both of the capturing times can be comparedwith each other on the time axis. In other words, according to thepresent example embodiment, it is possible to show the detection by theuser and the detection by the detection processing in comparison witheach other at what time during the examination the lesion to be capturedis detected. Therefore, it is possible to easily recognize thediscrepancy between the detection result of the detection processing forthe image captured by the endoscope and the detection result by theuser. For example, when a lesion image is obtained during one detectionand no lesion image is obtained during the other detection at a certaintime zone during the examination, it is possible to easily grasp thatthere is a discrepancy in detection results during the detection in sucha time zone.

The information processing device 1 includes a processor and a memory ascomponents which are not shown. The processor reads a computer program,in which the above-described processing of the information processingdevice 1 is implemented, from the memory, and executes the computerprogram. Thus, the processor realizes functions of the first acquisitionunit 2, the second acquisition unit 3, and the display control unit 4.

Alternatively, each of the first acquisition unit 2, the secondacquisition unit 3, and the display control unit 4 may be realized bydedicated hardware. Further, a part or all of components of each devicemay be realized by a general-purpose or dedicated circuitry, aprocessor, or a combination thereof. The components may be configured bya single chip, or may be configured by a plurality of chips connected toeach other via a bus. A part or all of components of each device may berealized by a combination of the circuitry and the program describedabove. In addition, as a processor, a CPU (Central Processing Unit) or aGPU (Graphics Processing Unit) can be used.

Further, when a part or all of the components of the informationprocessing device 1 are realized by a plurality of informationprocessing devices and circuits, the plurality of information processingdevices and the circuits may be arranged in a centralized manner, or maybe arranged in a distributed manner. For example, the informationprocessing devices and the circuits may be realized as a form of beingconnected to each other via a communication network such as aclient-and-server system or a cloud computing system. Further, thefunction of the information processing device 1 may be provided in SaaS(Software as a Service) format.

Example embodiments will be described below in which the first exampleembodiment is made more specific.

Second Example Embodiment

FIG. 2 is a block diagram showing a configuration of an examinationsupport system 10 according to a second example embodiment. Theexamination support system 10 includes an endoscope system 100, aninformation processing device 200, and a display device 300. Theendoscope system 100 is used to examine body cavities of a subject to beexamined. For example, the endoscope system 100 is used to examine alarge intestine, but may be used to examine other digestive tracts.

The endoscope system 100 includes an endoscope 110, a light sourcedevice 120, a processor device 130, and a display device 140. Theendoscope 110 is optically connected to the light source device 120, andfurther electrically connected to the processor device 130.

The endoscope 110 includes an insertion portion 111 (an insertion unit111) to be inserted into the body of a person who is a subject to beexamined and an operation portion 112 (an operation unit 112) configuredto operate a direction of a distal end of the insertion portion 111. Animage capturing portion 113 (an image capturing unit 113) is provided atthe endoscope 110 to capture an in-vivo image of the body. The imagecapturing portion 113 includes, for example, various lenses, an imagecapturing sensor, and a signal processing circuit. As the imagecapturing sensor, a sensor such as a CCD (Charge Coupled Device) or aCMOS (Complementary Metal-Oxide Semiconductor) is used. The variouslenses and the image capturing sensor are disposed at the distal end ofthe insertion portion 111, for example, and the other signal processingcircuits are disposed at the operation portion 112, for example. Theimage capturing portion 113 outputs an image signal of the capturedimage to the processor device 130 under the control of the processordevice 130.

A light guide is provided inside the insertion portion 111 to propagateillumination light from the light source device 120 to the distal end ofthe insertion portion 111, and the inside of the body can be illuminatedby the illumination light. Further, the insertion portion 111 isprovided with a treatment-instrument insertion passage through which atreatment instrument such as electrocautery is guided from the operationportion 112 to the distal end of the insertion portion 111. Therefore,the user (doctor) can excise the lesion site with the treatmentinstrument while looking at the image captured by the endoscope 110. Inaddition, the insertion portion 111 is provided with a nozzle forejecting air or water from the distal end of the insertion portion 111.

The light source device 120 is a device that supplies the illuminationlight to the above-described light guide provided in the endoscope 110under the control of the processor device 130. The illumination lightoutput from the light source device 120 is emitted from the distal endof the endoscope 110 by passing through the light guide. Thus, anin-vivo observation site is irradiated with the illumination light.

The processor device 130 is electrically connected to the endoscope 110,the light source device 120, the display device 140, and the informationprocessing device 200. Further, the processor device 130 is connected toan input device that receives an input of an instruction from the user.Particularly, in the present embodiment, the input device receives, asan input, an instruction to save the image captured by the endoscope 110during the examination with the endoscope 110. Such an input device maybe provided as a function of the operation portion 112, or when thedisplay device 140 is configured by a touch panel, the touch panel mayfunction as such an input device. An input device such as a keyboard ora mouse provided independently of the above-described configuration maybe used.

The processor device 130 comprehensively controls an operation of theendoscope system 100. Particularly, the processor device 130 performspredetermined image processing on the image signal received from theendoscope 110, and generates a captured image to be displayed on thedisplay device 140. Further, as shown in FIG. 3, the processor device130 generates a display image 52 in which a captured image is arrangedin a captured image area 50 and characters or images of variousreference information such as information on the subject to be examinedare arranged in a non-captured image area 51. The display image 52 is animage displayed on the entire screen of the display device 140. Theprocessor device 130 controls the display device 140 to display thedisplay image 52. In addition, the processor device 130 outputs thedisplay image 52 to the information processing device 200. The processordevice 130 may output the captured image to the information processingdevice 200. In this way, the processor device 130 outputs the displayimage 52 or the captured image to the information processing device 200,and thus sequentially outputs the current image captured by theendoscope 110 to the information processing device 200. In other words,the processor device 130 outputs the moving image captured by theendoscope 110, that is, a series of images of the body cavity capturedcontinuously in time to the information processing device 200 in realtime. In the present embodiment, the processor device 130 outputs theimage to the information processing device 200 in real time in order todetect the lesion site in real time during the examination in theinformation processing device 200 to be described below, but does notnecessarily output the image in real time. In other words, when thelesion site is not necessary to be detected in real time by theinformation processing device 200, the processor device 130 does nothave to output the captured image in real time. For example, thecaptured image that is captured during the examination may be output tothe information processing device 200 at any time after the examinationis completed and a series of capturing is completed. In this way, theprocessor device 130 functions as an image output device that outputsthe image captured by the endoscope 110.

Further, upon receiving, as an input, the instruction to save thecaptured image during the capturing with the endoscope 110, theprocessor device 130 saves image data of the captured image at thetiming of receiving the input in a storage device such as a memory. Theuser regards the captured image showing the lesion site as a target tobe saved, for example. Therefore, when the user inputs the instructionto save the lesion image during the capturing with the endoscope 110,the processor device 130 saves the image data of captured image at thetiming when the input is received. In other words, the processor device130 saves the lesion image that user has instructed to save out of theseries of images captured by the endoscope 110 during the examinationwith the endoscope 110.

In the present embodiment, when the user instructs to save the lesionimage during the examination with the endoscope 110, the processordevice 130 outputs an image with a predetermined feature until a certaintime elapses from the time of the instruction. Specifically, forexample, the processor device 130 outputs the captured image to be saveduntil a certain time elapses from the time of the instruction. In otherwords, during this period, the processor device 130 continuously outputsthe same image. In other words, the processor device 130 outputs animage having a feature of being the same as the previous image duringthis period. Thus, a frozen image is output to the display device 140and the information processing device 200, as an image of a series ofimages during this period. By such processing, the display device 140displays the image having a predetermined feature until a certain timeelapses from the time of the instruction. Therefore, the user can graspthat the captured image has been saved by the processor device 130 inresponse to his/her instruction.

The predetermined feature is not limited to the above-described feature,and may be another feature. For example, until a certain time elapsesfrom the time of the instruction, the display image 52 arranged with notonly the captured image but also a thumbnail image of the image to besaved may be output to the display device 140 and the informationprocessing device 200. In this case, during this period, the processordevice 130 outputs an image having a feature that the thumbnail image isarranged in a predetermined area.

The processor device 130 includes, for example, a memory and a processorsuch as a CPU and a GPU, and the processor reads software (computerprogram) including one or more commands from the memory and executes thesoftware to realize the processing of the processor device 130.

The display device 140 displays the display image 52 generated by theprocessor device 130. Specifically, the display device 140 is a flatpanel display such as a liquid crystal display, a plasma display, or anorganic EL (Electro-Luminescence) display.

Next, the information processing device 200 and the display device 300will be described.

The display device 300 is electrically connected to the informationprocessing device 200, and is a device that displays an image under thecontrol of the information processing device 200. Specifically, thedisplay device 300 is a flat panel display such as a liquid crystaldisplay, a plasma display, or an organic EL display. In the presentembodiment, the display device 300 is configured as a touch panel, andalso functions as an input device that receives the input from the user.

The information processing device 200 corresponds to the informationprocessing device 1 shown in FIG. 1, and is a device that supports theexamination by the user (for example, a doctor). Details of theinformation processing device 200 will be described below.

FIG. 4 is a block diagram showing an example of a functionalconfiguration of the information processing device 200. As shown in FIG.4, the information processing device 200 includes an output imageacquisition unit 210, a lesion detection unit 220, an operation logacquisition unit 230, a detection log acquisition unit 240, an imageselection unit 250, and an input/output control unit 260.

The output image acquisition unit 210 acquires an image group (eachframe image (still image) constituting the captured moving image) outputby the processor device 130. In other words, the output imageacquisition unit 210 acquires a series of images captured by theendoscope 110 during the examination with the endoscope 110. Morespecifically, the output image acquisition unit 210 acquires, from theprocessor device 130, a series of images of the body cavity capturedcontinuously in time. In the present embodiment, the output imageacquisition unit 210 sequentially acquires current images captured bythe endoscope 110 in real time, but does not necessarily acquire theimages in real time. When the display image 52 is output from theprocessor device 130, the output image acquisition unit 210 performsprocessing of cutting out the captured image arranged in the capturedimage area 50 from the display image 52, thereby obtaining the capturedimage used for the processing of the lesion detection unit 220.

The lesion detection unit 220 sequentially performs detection processingof a lesion site on the image acquired by the output image acquisitionunit 210. In other words, the lesion detection unit 220 sequentiallyperforms detection processing of a lesion site on each frame imageconstituting the captured moving image. In the present embodiment, thelesion detection unit 220 performs the detection processing of thelesion site in real time. For example, the lesion detection unit 220preferably performs the detection processing at a processing speedfaster than a frame rate of the captured moving image. The lesiondetection unit 220 detects the lesion site from the image by performingknown image recognition processing. In such detection processing, aposition of the lesion site in the image is also detected. For example,the lesion detection unit 220 performs the detection processing of thelesion site by inputting the image, which is acquired by the outputimage acquisition unit 210, to a model learned in advance by a machinelearning algorithm. Such a model is, for example, a model learned bydeep learning such as CNN (Convolution Neural Network), but may be amodel learned by using another machine learning algorithm. For example,based on whether an index value (accuracy) indicating the probabilitythat the lesion site is depicted in an image output from the modeldescribed above exceeds a predetermined threshold value, the lesiondetection unit 220 determines whether the lesion site is depicted in theimage.

When detecting the lesion site, the lesion detection unit 220 stores thelesion image, position information in the image of the detected lesionsite, the index value described above, and information indicating thecapturing time of the lesion image (hereinafter, referred to ascapturing time information) in a storage device such as a memory 291,which will be described below, as a log. Here, the capturing timeinformation may be any information that can specify the time of thecapturing. For example, the capturing time information is a system timeat the time of the capturing. As described above, the captured image isacquired by the output image acquisition unit 210 in real time, and thelesion detection processing is performed in real time. Therefore, thelesion detection unit 220 may use the time when the informationprocessing device 200 acquires the image, as capturing time informationof the image, or may use the time when the lesion detection unit 220performs the lesion detection processing, as capturing time informationof the image. In addition, the capturing time information may beincidental information incidental to the image data by the processordevice 130. As described above, the capturing time information may beany information that can specify the time when the capturing isperformed, and thus the frame number of the moving image may be usedinstead of the system time.

The operation log acquisition unit 230 corresponds to the firstacquisition unit 2 shown in FIG. 1. The operation log acquisition unit230 acquires a capturing time of a lesion image that the user hasinstructed to save, from a series of images captured by an endoscope 110in the midst of examination with the endoscope 110. Further, theoperation log acquisition unit 230 acquires the lesion image instructedto be saved by the user. In the following description, the acquiredcapturing time and the lesion image may be referred to as a log.

In the present embodiment, the operation log acquisition unit 230acquires the capturing time of the lesion image instructed to be savedby the user and the lesion image by analyzing a series of frame image(that is, moving images) output by the processor device 130.Specifically, the operation log acquisition unit 230 determines, basedon the presence or absence of a predetermined feature in the acquiredcaptured image, whether the capturing time of the captured image is thetime when the instruction to save is generated. When the predeterminedfeature exists in the acquired captured image, the operation logacquisition unit 230 determines that the instruction to save is input atthe capturing time of the captured image. Then, the operation logacquisition unit 230 acquires the captured image as a lesion imageinstructed to be saved by the user, and acquires the capturing time ofthe captured image as the capturing time of the lesion image instructedto be saved by the user. The operation log acquisition unit 230 mayacquire a time point around a predetermined time from the capturing timeof the captured image, as the capturing time of the lesion imageinstructed to be saved by the user. In this way, the operation logacquisition unit 230 determines, based on the presence or absence of thepredetermined feature in the image acquired by the output imageacquisition unit 210, the time point at which the instruction to save isgenerated, and thus acquires the capturing time of the lesion imageinstructed to be saved by the user and the lesion image.

As described above, in the present embodiment, when the user instructsto save the lesion image during the examination with the endoscope 110,the processor device 130 outputs a frozen image as a captured imageuntil a certain time elapses from the instruction time. In the presentembodiment, accordingly, the operation log acquisition unit 230determines the time point at which the instruction to save is generated,based on whether the image to be determined is the same as the image ofa previous frame, and thus acquires the capturing time of the lesionimage instructed to be saved by the user and the lesion image. When theprocessor device 130 outputs an image having another feature as apredetermined feature until a certain time elapses from the instructiontime, the operation log acquisition unit 230 may determine the presenceor absence of such another feature. For example, the operation logacquisition unit 230 may determine whether a thumbnail image is arrangedin the image acquired by the output image acquisition unit 210.

According to such processing of the operation log acquisition unit 230,the lesion image instructed to be saved by the user and the informationindicating the capturing time of the lesion image (capturing timeinformation) are acquired without acquisition of the log data from theprocessor device 130. The lesion image acquired by the operation logacquisition unit 230 is an image presumed to be the lesion imageinstructed to be saved by the user as described above. In other words,the operation log acquisition unit 230 searches for an image presumed tobe the lesion image instructed to be saved by the user, from the imagegroup acquired by the output image acquisition unit 210, and acquiresthe image. Further, similarly to the capturing time information storedby the lesion detection unit 220, the capturing time informationacquired by the operation log acquisition unit 230 may be, for example,the system time at the time point when the capturing is performed. Inthis case, the operation log acquisition unit 230 may use the time whenthe output image acquisition unit 210 acquires the image in real time,as the capturing time information of the image, or may use the time whenthe lesion detection unit 220 performs the lesion detection processingin real time, as capturing time information of the image. In addition,the capturing time information acquired by the operation log acquisitionunit 230 may be incidental information incidental to the image data bythe processor device 130. As the capturing time information, the framenumber of the moving image may be used instead of the system time.

Although the operation log acquisition unit 230 acquires, from theprocessing described above, the information regarding the operation ofthe instruction to save during the examination, the log data may beacquired from the processor device 130. In other words, the operationlog acquisition unit 230 may acquire the capturing time of the lesionimage instructed to be saved by the user and the lesion image which areoutput as log data by the processor device 130. In this case, theoperation log acquisition unit 230 may not perform the above-describeddetermination processing of determining, based on the presence orabsence of the predetermined feature, the time point at which theinstruction to save is generated.

The detection log acquisition unit 240 corresponds to the secondacquisition unit 3 shown in FIG. 1. The detection log acquisition unit240 acquires a capturing time of a lesion image detected by detectionprocessing on a series of images captured by an endoscope 110 in themidst of examination with the endoscope 110. Further, the detection logacquisition unit 240 acquires the lesion image detected by the detectionprocessing. In the present embodiment, the detection log acquisitionunit 240 acquires the lesion image and the capturing time information ofthe lesion image which are stored in the storage device as log data bythe lesion detection unit 220, from the storage device.

The image selection unit 250 specifies a plurality of lesion images inwhich the same lesion site is detected by the detection processing ofthe lesion detection unit 220, and selects one image (hereinafter,referred to as a representative image) from the plurality of specifiedlesion images. In the present embodiment, the image selection unit 250specifies a plurality of lesion images, in which the same lesion site isdetected, from the lesion image group acquired by the detection logacquisition unit 240, and selects a representative image from theplurality of lesion images. For example, the image selection unit 250compares the lesion images with each other to specify a plurality oflesion images in which the same lesion site is detected. Morespecifically, the image selection unit 250 performs object trackprocessing using feature points of the lesion site on continuouscaptured images (frame image) constituting the captured moving imageamong the lesion images acquired by the detection log acquisition unit240. Thus, the image selection unit 250 specifies a plurality of lesionimages in which the same lesion site is detected. By such processing, aplurality of images are specified in which the same lesion site isdepicted, the plurality of images being obtained when the same lesionsite is continuously captured in time. Then, the image selection unit250 selects a representative image from the plurality of lesion imagesin which the same lesion site is detected. The image selection unit 250selects, for example, a lesion image having the highest detectionaccuracy in the detection processing from these plurality of lesionimages. When there are a plurality of lesion images having the highestdetection accuracy in the detection processing, the image selection unit250 may select, from these lesion images, an image in which the lesionsite can be easily seen. For example, the image selection unit 250 mayselect an image of which position in the image of the lesion site isclosest to the center, as the image in which the lesion site is easilyseen, or may select an image in which the contrast between the lesionsite and the non-lesion site is maximum. The image selection unit 250may specify a plurality of lesion images, in which the same lesion siteis detected, by processing of calculating the similarity of imagesinstead of the object track processing. In this case, it is possible tospecify a plurality of images in which the same lesion site is depictedeven when the same lesion site is not continuously captured in time.

The image selection unit 250 may select a plurality of images asrepresentative images. Particularly, the image selection unit 250 mayselect two or more images having different capturing conditions from theplurality of lesion images in which the same lesion site is detected.For example, the image selection unit 250 may select two or more imageshaving different capturing angles, or may select two or more imageshaving different capturing distances. Thus, various images will bedisplayed as representative images.

The input/output control unit 260 controls input and output ofinformation in the information processing device 200. The informationprocessing device 200 is connected to an input device that receives aninput of an instruction from the user, and the input/output control unit260 receives the input from the input device. Particularly, in thepresent embodiment, the input device receives an input from a user whosees the display of the display device 300. In the present embodiment,the display device 300 is configured as a touch panel, and the displaydevice 300 also functions as an input device. An input device such as akeyboard or a mouse provided independently of the display device 300 maybe used.

In addition, the input/output control unit 260 controls the display ofthe display device 300. The input/output control unit 260 corresponds tothe display control unit 4 shown in FIG. 1. Therefore, in particularly,the input/output control unit 260 controls the display device 300 todisplay the capturing time acquired by the operation log acquisitionunit 230 and the capturing time acquired by the detection logacquisition unit 240 which are plotted on a time axis. In other words,the input/output control unit 260 controls to display the capturing timeof the lesion image saved by the instruction from the user and thecapturing time of the lesion image in which the lesion is detected bythe lesion detection unit 220 which are plotted on a time axis. In thepresent embodiment, the input/output control unit 260 further displaysthe lesion image in association with the capturing time. In other words,the input/output control unit 260 displays the lesion image saved by theinstruction from the user in association with the capturing time of thelesion image. Similarly, the input/output control unit 260 displays thelesion image in which the lesion is detected by the lesion detectionunit 220 in association with the capturing time of the lesion image.More specifically, as shown in FIG. 5 and the like to be describedbelow, the input/output control unit 260 displays a thumbnail image ofthe lesion image.

FIG. 5 is a schematic diagram showing a display example of the displaydevice 300 based on the control of the input/output control unit 260.Display control of the input/output control unit 260 of the presentembodiment will be described with reference to FIG. 5.

The input/output control unit 260 displays a list of examination capableof displaying logs in a log list area 60 on a screen of the displaydevice 300. In the example shown in FIG. 5, three kinds of examinationperformed on Jul. 19, 2019 are listed. In the example shown in FIG. 5,as each item in the list, a time is displayed during which theexamination is performed. In other words, a start time of theexamination and an end time of the examination are displayed. Theinput/output control unit 260 performs the display as shown in FIG. 5 bysetting a capturing time of the first image of the series of imagesacquired by the output image acquisition unit 210 as a start time ofexamination and specifying a capturing time of the last image of theseries of images as an end time of examination, for example.

The input/output control unit 260 displays, in a log display area 61 onthe screen of the display device 300, the information acquired by theoperation log acquisition unit 230 and the detection log acquisitionunit 240, when the input is received from the user to select one of theitems from the list displayed in the log list area 60. In the exampleshown in FIG. 5, the log for the examination performed at theexamination time from 14:00 to 14:33 is displayed.

The input/output control unit 260 displays, in the log display area 61,a time axis 62, a mark 63 indicating the capturing time acquired by theoperation log acquisition unit 230 on the time axis 62, and a mark 64indicating the capturing time acquired by the detection log acquisitionunit 240 on the time axis 62. Further, the input/output control unit 260displays a lesion image corresponding to the capturing time acquired bythe operation log acquisition unit 230 and a lesion image correspondingto the capturing time acquired by the detection log acquisition unit240. In more detail, a thumbnail image of the lesion image is displayed.In the example shown in FIG. 5, lesion images 65 a and 65 b aredisplayed as lesion images corresponding to the capturing time acquiredby the operation log acquisition unit 230, and lesion images 66 a, 66 b,and 66 c are displayed as lesion images corresponding to the capturingtime acquired by the detection log acquisition unit 240. In the exampleshown in FIG. 5, the marks 63 and 64 are arrows as an example. In theexample shown in FIG. 5, a display related to the information acquiredby the operation log acquisition unit 230 is performed on an upper sideof the time axis 62. In other words, a display related to the lesionimage saved by the user is performed on the upper side of the time axis62. The input/output control unit 260 displays the lesion image acquiredby the operation log acquisition unit 230 as the lesion image detectedby the user (for example, a doctor). On the other hand, a displayrelated to the information acquired by the detection log acquisitionunit 240 is performed on a lower side of the time axis 62. In otherwords, a display related to the lesion automatically detected by thedetection processing performed by the lesion detection unit 220 isperformed on the lower side of the time axis 62. The example shown inFIG. 5 is merely an example, and it goes without saying that the displaycontent on the upper side and the display content on the lower side ofthe time axis 62 may be reversed. In the example shown in FIG. 5, thetime axis 62 extends in a horizontal direction of the screen, but thetime axis 62 may extend in another direction.

In the example shown in FIG. 5, the input/output control unit 260displays scales of the time axis 62, but the scales of the time axis 62may not be displayed. In the example shown in FIG. 5, the start time ofthe examination is displayed as a scale at a beginning end of the timeaxis 62, and the end time of the examination is displayed as a scale ata terminated end of the time axis 62, but the scale is not necessarilyassociated with the examination time. For example, a value of the scaleat the beginning end of the time axis 62 may be set to 0, and a value ofthe scale at the terminated end of the time axis 62 may be a valueequivalent to a time length of the captured image.

The input/output control unit 260 may display the position in the imageof the lesion site detected by the lesion detection unit 220. In theexample shown in FIG. 5, specifically, the input/output control unit 260displays a frame 67 surrounding the lesion site.

Further, as shown in FIGS. 5 to 7, the input/output control unit 260displays, based on the instruction input from the user, the scale of thetime axis 62 in an enlarged or reduced manner. The input/output controlunit 260 may change the scale of the time axis 62 when an operation oftouching (tapping) or clicking the time axis 62 is performed as such aninstruction, for example. Thereby, it is possible to improve theconvenience in browsing the log.

FIG. 6 shows a display example when the scale of the time axis 62 shownin FIG. 5 is enlarged. In the example shown in FIG. 6, the scale of thetime axis 62 is enlarged as compared with the example shown in FIG. 5,and a log of a time zone from 14:15 to 14:18 is displayed as anexamination time. In addition, FIG. 7 shows a display example when thescale of the time axis 62 shown in FIG. 6 is enlarged. In the exampleshown in FIG. 7, the scale of the time axis 62 is enlarged as comparedwith the example shown in FIG. 6, and a log of a time zone from 14:16:20to 14:16:50 is displayed as an examination time.

The input/output control unit 260 may change a log actually displayedamong the logs of the time zone to be displayed according to the scaleof the time axis 62. As shown in FIG. 5, in the display where the scaleis reduced, there is a concern that the display may be complicated whenall the logs belonging to the time zone to be displayed (specifically,from 14:00 to 14:33 in FIG. 5) are displayed. Therefore, for example,when the time axis 62 is displayed with a scale smaller than apredetermined scale, the input/output control unit 260 may display onlysome of the logs of the time zone to the displayed. In other words, theinput/output control unit 260 may display only some of the capturingtimes and the captured image corresponding to the capturing time. Insuch a case, as for the capturing time, all the capturing time in thetime zone to be displayed may be displayed, and only the display of thecaptured image may be limited to some of the captured images. During theselection of some of the captured images, the input/output control unit260 may select it according to an arbitrary selection criterion. Forexample, the input/output control unit 260 may select the image based onthe magnitude of the accuracy of detection in the detection processing.

The display of some of the logs described above will be described indetail with reference to the drawings. In the example shown in FIG. 6,lesion images 66 a, 66 d, and 66 b are displayed as logs of thedetection processing of the lesion detection unit 220 in the time zonefrom 14:15 to 14:18. On the other hand, in the example shown in FIG. 5,focusing on the same time zone as in FIG. 6, only the lesion images 66 aand 66 b are displayed as the logs of the detection processing of thelesion detection unit 220. Similarly, in the example shown in FIG. 7,lesion images 66 d, 66 b, and 66 e are displayed as logs of thedetection processing of the lesion detection unit 220 in the time zonefrom 14:16:20 to 14:16:50. On the other hand, in the example shown inFIG. 6, focusing on the same time zone as in FIG. 7, only the lesionimages 66 d and 66 b are displayed as the logs of the detectionprocessing of the lesion detection unit 220.

Further, the input/output control unit 260 may display only the lesionimage selected as a representative image by the above-described imageselection unit 250, from the plurality of lesion images in which thesame lesion site is detected. In other words, the lesion image may notbe displayed which is not selected as the representative image from theplurality of lesion images in which the same lesion site is detected. Atthis time, the input/output control unit 260 may display the capturingtime of the lesion image selected as the representative image, and maynot display the capturing time of the lesion image not selected as therepresentative image from the plurality of lesion images in which thesame lesion site is detected. Thereby, the repeated lesions are notdisplayed, and visibility is improved. In the present embodiment, whenthe time axis 62 is displayed with a scale smaller than a predeterminedscale, the input/output control unit 260 displays only the lesion imageselected as the representative image from the plurality of lesion imagesin which the same lesion site is detected. A specific example thereofwill be described with reference to the drawings. In the example shownin FIG. 7, the lesion image 66 b and the lesion image 66 e are displayedin which the same lesion site is captured, whereas in the example shownin FIG. 6, only the lesion image 66 b is displayed which is selected asthe representative image from these captured images.

An example of a hardware configuration of the information processingdevice 200 will be described below. FIG. 8 is a schematic diagramshowing an example of a hardware configuration of the informationprocessing device 200. As shown in FIG. 8, the information processingdevice 200 includes an input/output interface 290, a memory 291, and aprocessor 292.

The input/output interface 290 is an input/output circuit configured tocommunicate with any other devices, for example, the processor device130 and the display device 300.

The memory 291 is configured by a combination of a volatile memory and anon-volatile memory, for example. The memory 291 is used to storesoftware (computer program) and data used for various processing of theinformation processing device 200, the software including one or morecommands executed by the processor 292.

The processor 292 reads and executes the software (computer program)from the memory 291 to perform the processing of each component shown inFIG. 4. Specifically, the processor 292 performs the processing of theoutput image acquisition unit 210, the lesion detection unit 220, theoperation log acquisition unit 230, the detection log acquisition unit240, the image selection unit 250, and the input/output control unit260.

The processor 292 may be, for example, a CPU or a GPU. The processor 292may include a plurality of processors.

As described above, the information processing device 200 has a functionas a computer.

The above-described programs may be stored and supplied to a computerusing various types of non-transitory computer readable media. Thenon-transitory computer readable media include various types of tangiblestorage media. Examples of the non-transitory computer readable mediainclude a magnetic recording medium (for example, a flexible disk, amagnetic tape, and a hard disk drive), a magneto-optic recording medium(for example, a magneto-optic disk), a CD-ROM (Read Only Memory), aCD-R, a CD-R/W, and a semiconductor memory (for example, a mask ROM, aPROM (Programmable ROM), an EPROM (Erasable PROM), a flash ROM, and aRAM (Random Access Memory)). These programs may be supplied to computersusing various types of transitory computer readable media. Examples ofthe transitory computer readable media include an electrical signal, anoptical signal, and an electromagnetic wave. The transitory computerreadable media can supply programs to a computer through a wiredcommunication line, for example, electric wires and optical fibers, or awireless communication line.

An operation example of the information processing device 200 will bedescribed below. FIG. 9 is a flowchart showing an example of theoperation of the information processing device 200. The operationexample will be described below with reference to the flowchart of FIG.9.

In step S100, the output image acquisition unit 210 acquires frameimages constituting the captured moving image output by the processordevice 130. After step S100, the process proceeds to steps S110 andS120. Processes from step S110 to step S112 and processes from step S120to step S121 are performed in parallel, for example, but may beperformed in order.

The processes from step S110 to step S112 will be described.

In step S110, the lesion detection unit 220 sequentially performsdetection processing of a lesion site on the image acquired in stepS110. The lesion detection unit 220 stores a log when the lesion site isdetected.

Next, in step S111, the detection log acquisition unit 240 acquires thelesion image and capturing time information of the lesion image storedas the log.

Next, in step S112, the image selection unit 250 specifies a pluralityof lesion images, in which the same lesion site is detected, withrespect to the lesion image acquired in step S111, and selects arepresentative image from the plurality of specified lesion images. Whenthe input/output control unit 260 does not display a summary of theplurality of lesion images in which the same lesion site is detected,such a step may be omitted.

The processes from step S120 to step S121 will be described.

In step S120, the operation log acquisition unit 230 analyzes the imageacquired in step S100 to specify a timing at which the user hasinstructed to save. In other words, the operation log acquisition unit230 specifies a generation timing of an operation of saving a stillimage (lesion image). Then, in step S121, the operation log acquisitionunit 230 acquires a log related to the operation of the instruction tosave performed during the examination. Specifically, the operation logacquisition unit 230 acquires, based on the timing specified in stepS120, the capturing time of the lesion image instructed to be saved bythe user and the lesion image.

After the processes of steps S112 and S121, the process proceeds to stepS130. In step S130, the input/output control unit 260 displays the logon the display device 300.

The second example embodiment has been described above. According to thepresent embodiment, the capturing time of the lesion image instructed tobe saved by the user during the examination with the endoscope and thecapturing time of the lesion image of the lesion detected by theinformation processing device 200 are plotted on the time axis togetherwith the lesion image. In other words, the capturing time of the lesionimage of the lesion detected by the user and the capturing time of thelesion image of the lesion automatically detected are plotted on thetime axis together with the lesion image. Therefore, both detectionsituations can be compared with each other on the time axis.Accordingly, it is possible to easily recognize the discrepancy betweenthe detection result by the automatic detection processing and thedetection result by the user.

Third Example Embodiment

A third example embodiment will be described below. For example, in theendoscopy of the large intestine, first, an endoscope 110 is inserted upto a beginning end side of the large intestine (an end side connected tothe small intestine). Thereafter, substantive examination is started. Inother words, observation is performed in order from the beginning endside of the large intestine toward a terminated end side of the largeintestine (an end side connected to the anus) with the endoscope 110. Inthis way, after the position of the endoscope 110 reaches the startposition of the examination, the actual examination is started. In otherwords, it is assumed that the user does not detect the lesion, that is,does not give an instruction to save the lesion image until the positionof the endoscope 110 reaches the start position of the examination. Onthe other hand, the lesion detection unit 220 can also detect the lesionimage from the image captured until the position of the endoscope 110reaches the start position of the examination. However, as describedabove, since the actual examination of the user is performed after theposition of the endoscope 110 reaches the start position of theexamination, the advantage of displaying the log on the display device300 until the position of the endoscope 110 reaches the start positionof the examination is insufficient.

Therefore, in the present embodiment, a description will be given withrespect to a configuration in which the log after the position of theendoscope 110 reaches the start position of the examination is displayedon the display device 300.

Hereinafter, differences from the second example embodiment will bedescribed, and duplicated descriptions will not be made. In the presentembodiment, an information processing device 201 is used instead of theinformation processing device 200.

FIG. 10 is a block diagram showing an example of a functionalconfiguration of the information processing device 201 according to thethird example embodiment. As shown in FIG. 10, the informationprocessing device 201 differs from the information processing device 200according to the second example embodiment in that a predetermined siteimage specifying unit 265 is further provided. Processing of thepredetermined site image specifying unit 265 is realized when theprocessor 292 reads and executes software (computer program) from thememory 291, for example.

The predetermined site image specifying unit 265 specifies an image, inwhich a predetermined site in the body cavity is captured, from a seriesof images captured by the endoscope 110 during the examination with theendoscope 110. For example, in the case of endoscopy of the largeintestine, the predetermined site may be a site existing on thebeginning end side of the large intestine. In this case, specifically,the predetermined site may be, for example, an ileocecal valve, or anentrance portion of the appendix. These sites are merely example, andthe predetermined site may be a site near the start position of theexamination. The predetermined site image specifying unit 265 performsthe known image recognition processing on the image captured by theendoscope 110, and specifies the image in which the predetermined siteis captured. For example, the predetermined site image specifying unit265 inputs an image to the model learned in advance by a machinelearning algorithm to determine whether the image is an image in whichthe predetermined site is captured. Such a model is, for example, amodel learned by deep learning such as CNN, but may be a model learnedusing another machine learning algorithm.

The input/output control unit 260 of the present embodiment displays, asa display target, the log after the capturing time of the image in whichthe predetermined site is captured. In other words, the input/outputcontrol unit 260 plots the capturing time after the capturing time ofthe image in which the predetermined site is captured on the time axis62. Similarly, the input/output control unit 260 displays a lesion imagecaptured after the capturing time of the image in which thepredetermined site is captured. In other words, the input/output controlunit 260 excludes the logs (that is, the capturing time of the lesionimage and the lesion image) up to the capturing time of the image inwhich the predetermined site is captured from the display target. Asdescribed above, it is assumed that the user does not give aninstruction to save the lesion image until the position of the endoscope110 reaches the start position of the examination, but the lesiondetection unit 220 can detect the lesion image regardless of whether theposition of the endoscope 110 has reached the start position of theexamination. Therefore, the input/output control unit 260 preferablyrestricts the display of the log of the detection processing by thelesion detection unit 220 as described above.

An operation example of the information processing device 201 will bedescribed below. FIG. 11 is a flowchart showing an example of theoperation of the information processing device 201. The flowchart shownin FIG. 11 differs from the flowchart shown in FIG. 9 in that steps S140and S141 are added. Differences from the flowchart shown in FIG. 9 willbe described.

The information processing device 201 further performs processes ofsteps S140 and S141 after the process of step S100. Processes from stepS110 to step S112, processes from step S120 to step S121, and processesfrom step S140 to step S141 are performed in parallel, for example, butmay be performed in order.

In step S140, the predetermined site image specifying unit 265 specifiesan image, in which a predetermined site is captured, from a series ofimages captured by the endoscope 110 during the examination with theendoscope 110.

Next, in step S141, the input/output control unit 260 specifies anactual start time of the examination based on the processing result instep S140. In other words, the input/output control unit 260 sets thecapturing time of the image, in which the predetermined site iscaptured, as the actual start time of the examination.

Then, in step S130, the input/output control unit 260 displays, as adisplay target, the log after the time specified in step S141.

The third example embodiment has been described above. According to thepresent embodiment, the information processing device 201 displays, asthe display target, the log after the time when the predetermined siteis captured. Therefore, only the log after the position of the endoscope110 reaches the start position of the actual examination can bedisplayed. In other words, it is possible to prevent the display of thelog before the start of the actual examination. Therefore, thevisibility of the display of the log of the actual examination can beimproved.

Fourth Example Embodiment

A fourth example embodiment will be described below. In theabove-described embodiments, the discrepancy between the detectionresult by the detection processing on the captured image of theendoscope 110 and the detection result by the user is found by the userwho has seen the display of the display device 300. In the presentembodiment, a configuration for automatically detecting such discrepancywill be described.

Hereinafter, differences from the second example embodiment will bedescribed, and duplicated descriptions will not be made. In the presentembodiment, an information processing device 202 is used instead of theinformation processing device 200.

FIG. 12 is a block diagram showing an example of a functionalconfiguration of the information processing device 202 according to thefourth example embodiment. As shown in FIG. 12, the informationprocessing device 202 differs from the information processing device 200according to the second example embodiment in that a failure detectionunit 270 is further provided. Processing of the failure detection unit270 is realized when the processor 292 reads and executes software(computer program) from the memory 291, for example.

The failure detection unit 270 detects a failure of detection of alesion site by the user or the detection processing. In other words, thefailure detection unit 270 detects a failure of detection of the lesionsite by the user during the examination with the endoscope 110, or afailure of detection of the lesion site by the lesion detection unit220. Here, the failure of detection includes a failure due to omissionof detection of the lesion site and a failure due to erroneous detectionof a normal site as a lesion site (that is, a failure due to erroneousdetection). The failure detection unit 270 detects a failure by checkinga corresponding relation between the log acquired by the operation logacquisition unit 230 and the log acquired by the detection logacquisition unit 240. The failure detected by the failure detection unit270 may not actually be a failure. Therefore, it can be said that thefailure detection unit 270 has detected failure candidates.

The failure detection unit 270 of the present embodiment detects thefailure of detection of the lesion site by the user or the detectionprocessing by comparing the lesion image acquired by the operation logacquisition unit 230 with the lesion image acquired by the detection logacquisition unit 240. Specifically, the failure detection unit 270determines whether the detection log acquisition unit 240 acquires alesion image depicted with the same lesion site as the lesion sitedepicted in the lesion image acquired by the operation log acquisitionunit 230. Similarly, the failure detection unit 270 determines whetherthe operation log acquisition unit 230 acquires a lesion image depictedwith the same lesion site as the lesion site depicted in the lesionimage acquired by the detection log acquisition unit 240. The failuredetection unit 270 calculates, for example, similarity between thelesion image acquired by the operation log acquisition unit 230 and thelesion image acquired by the detection log acquisition unit 240, andthus determines whether the lesion site depicted in one lesion image isdepicted in the other lesion image.

When the detection log acquisition unit 240 does not acquire the lesionimage depicted with the same lesion site as the lesion site depicted inthe lesion image acquired by the operation log acquisition unit 230,there is a possibility that omission of detection by the lesiondetection unit 220 or erroneous detection by the user has occurred.Therefore, the failure detection unit 270 detects the failure ofdetection of the lesion site when the detection log acquisition unit 240does not acquire the lesion image depicted with the same lesion site asthe lesion site depicted in the lesion image acquired by the operationlog acquisition unit 230. In other words, the failure detection unit 270detects that a failure has occurred in the lesion image depicted withthe lesion site.

Further, when the operation log acquisition unit 230 does not acquirethe lesion image depicted with the same lesion site as the lesion sitedepicted in the lesion image acquired by the detection log acquisitionunit 240, there is a possibility that omission of detection by the useror erroneous detection by the lesion detection unit 220 has occurred.Therefore, the failure detection unit 270 detects the failure ofdetection of the lesion site when the operation log acquisition unit 230does not acquire the lesion image depicted with the same lesion site asthe lesion site depicted in the lesion image acquired by the detectionlog acquisition unit 240. In other words, the failure detection unit 270detects that a failure has occurred in the lesion image depicted withthe lesion site.

When the failure detection unit 270 detects the failure, theinput/output control unit 260 of the present embodiment displays adisplay indicating which lesion image the failure has been detected.FIG. 13 shows a display example of the display device 300 when thefailure detection unit 270 detects a failure. In the example shown inFIG. 13, it is displayed that failures have been detected with respectto a lesion image 66 b, a lesion image 65 b, and a lesion image 66 c.

In the above description, the failure detection unit 270 detects thefailure by comparing he lesion images, but the failure detection unit270 may detect the failure by comparing the capturing times. Forexample, the failure detection unit 270 may detect the failure ofdetection of the lesion site by the user or the detection processing bycomparing the capturing time acquired by the operation log acquisitionunit 230 with the capturing time acquired by the detection logacquisition unit 240. In other words, the failure detection unit 270 maydetect the failure by comparing the capturing time of the lesion imageinstructed to be saved by the user with the capturing time of the lesionimage detected by the detection processing. Specifically, the failuredetection unit 270 checks whether only any one of the capturing time ofthe lesion image instructed to be saved by the user and the capturingtime of the lesion image detected by the detection processing is notacquired every predetermined unit time.

When only the capturing time of the lesion image instructed to be savedby the user is acquired in the unit time of the check target, there is apossibility that omission of detection by the lesion detection unit 220or erroneous detection by the user may occur. Accordingly, in this case,the failure detection unit 270 detects the failure of detection of thelesion site captured in the unit time. In addition, when only thecapturing time of the lesion image detected by the detection processingis acquired in the unit time of the check target, there is a possibilitythat omission of detection by the user or erroneous detection by thelesion detection unit 220 may occur. Accordingly, in this case, thefailure detection unit 270 detects the failure of detection of thelesion site captured in the unit time. When the failure detection unit270 detects the failure by comparing the capturing times, theinput/output control unit 260 may display a display indicating whichcapturing time the failure has been detected.

In this way, the lesion detection unit 220 may detect the failure bycomparing the lesion images, or may detect the failure by comparing thecapturing times. However, it is preferable to detect the failure bycomparing the lesion images from the following reasons. In the case ofcomparing the capturing times, even when the user and the lesiondetection unit 220 appropriately detect the same lesion site, it will bedetermined that the failure of detection has occurred when there is notemporal corresponding relation between the lesion images. On the otherhand, in the case of comparing the lesion images, since the imagecontents are compared with each other, the occurrence of such adetermination result can be prevented.

An operation example of the information processing device 202 will bedescribed below. FIG. 14 is a flowchart showing an example of theoperation of the information processing device 202. The flowchart shownin FIG. 14 differs from the flowchart shown in FIG. 9 in that step S150is added and step S130 is replaced with step S151. Differences from theflowchart shown in FIG. 9 will be described below.

In the flowchart shown in FIG. 14, after processes of steps S112 andS121, the process proceeds to step S150. In step S150, the failuredetection unit 270 detects the failure of detection of the lesion siteby the user or the detection processing by checking a correspondingrelation between the log acquired by the operation log acquisition unit230 and the log acquired by the detection log acquisition unit 240.After step S150, the process proceeds to step S151. In step S151, theinput/output control unit 260 displays the log and the failure detectionresult on the display device 300.

The fourth example embodiment has been described above. According to thepresent embodiment, the failure of detection of the lesion site isdetected by checking the corresponding relation between the log acquiredby the operation log acquisition unit 230 and the log acquired by thedetection log acquisition unit 240. For this reason, the user can moreeasily grasp the discrepancy between the detection result by thedetection processing on the captured image of the endoscope 110 and thedetection result by the user.

Fifth Example Embodiment

A fifth example embodiment will be described below. In the presentembodiment, a configuration will be described in which feedback data isgenerated to improve detection accuracy of the lesion detection unit220.

Hereinafter, differences from the second example embodiment will bedescribed, and duplicated descriptions will not be made. In the presentembodiment, an information processing device 203 is used instead of theinformation processing device 200.

FIG. 15 is a block diagram showing an example of a functionalconfiguration of the information processing device 203 according to thefifth example embodiment. As shown in FIG. 15, the informationprocessing device 203 differs from the information processing device 200according to the second example embodiment in that a feedback datageneration unit 275 is further provided. Processing of the feedback datageneration unit 275 is realized when the processor 292 reads andexecutes software (computer program) from the memory 291, for example.

The feedback data generation unit 275 generates feedback data based onthe information input from the user. Specifically, the feedback datageneration unit 275 generates, as feedback data, data in which thelesion image is associated with detection failure information. Here,detection failure information is analysis information about the failureof detection by the lesion detection unit 220, and includes the type ofdetection failure (failure due to detection omission of the lesion orfailure due to erroneous detection of the lesion) and the cause of thefailure. However, the detection failure information may not necessarilyhave to include the cause of the failure. The feedback data generationunit 275 generates feedback data based on the instruction to select thelesion image input from the user and the detection failure informationon the selected lesion image input from the user.

In the present embodiment, the input/output control unit 260 furtherprovides a user interface for generating feedback data. In other words,the input/output control unit 260 performs processing of receiving theinstruction to select the lesion image as an input and processing ofreceiving the detection failure information on the selected lesion imageas an input. In addition, the input/output control unit 260 displays ascreen for such processing.

FIGS. 16 and 17 are schematic diagrams showing a display example of ascreen displayed on the display device 300 for generating feedback data.The input/output control unit 260 performs display for inputting thedetection failure information in a failure information area 70 on thescreen of the display device 300. The input/output control unit 260displays a screen to input the detection failure information whenreceiving an input from the user to select one of the lesion imagesdisplayed on the log display screen shown in FIG. 5. The selection ofthe lesion image may be a selection of the lesion image acquired by thedetection log acquisition unit 240, or may be a selection of the lesionimage acquired by the operation log acquisition unit 230. For example,the lesion image is selected when the user performs an operation oftouching (tapping) or clicking of any lesion image.

In the failure information area 70, a selected lesion image 71 isdisplayed, and an input field 72 of the detection failure information isdisplayed. The input field 72 includes a type input field 721 which isan input field for the type of detection failure and a cause input field722 which is an input field for the cause of the detection failure. Theuser looks at the lesion image 71 and inputs the detection failureinformation. For example, the user selects an appropriate type ofdetection failure from options, and selects an appropriate cause of thefailure from the options.

FIG. 16 shows an example of an input screen of detection failureinformation on the lesion image depicted with a lesion erroneouslydetected by the lesion detection unit 220. In the example shown in FIG.16, examples of options for the cause of erroneous detection includeerroneous detection due to reflection of an instrument, erroneousdetection due to blurring of the captured image, erroneous detection dueto detection of bubbles as a lesion, and erroneous detection due to acaptured image which is too darkened.

FIG. 17 shows an example of an input screen of detection failureinformation on the lesion image depicted with a lesion detected by theuser although the detection is overlooked by the lesion detection unit220. In FIG. 17, examples of options for the cause of detection omissioninclude detection omission due to reflection of the instrument,detection omission due to blurring of the captured image, detectionomission due to reflection of bubbles or the like, and detectionomission due to a flat lesion that is presumed to make detectiondifficult.

The feedback data generation unit 275 generates, as feedback data, datain which the selected lesion image and the input detection failureinformation are associated with each other. Even for a lesion imagesimilar to the selected lesion image, the feedback data generation unit275 may generate feedback data associated with the detection failureinformation input for the selected lesion image. The feedback datageneration unit 275 may store the generated feedback data in the storagedevice such as the memory 291, or may provide the generated feedbackdata to another device. Further, the input/output control unit 260 maydisplay the generated feedback data as a history on the display device300.

An operation example of the information processing device 203 will bedescribed below. FIG. 18 is a flowchart showing an example of theoperation of the information processing device 203. The flowchart shownin FIG. 18 differs from the flowchart shown in FIG. 9 in that steps S160to S162 are added. Differences from the flowchart shown in FIG. 9 willbe described.

In the flowchart shown in FIG. 18, after a process of step S130, theprocess proceeds to step S160. In step S160, the input/output controlunit 260 receives an input for selecting a lesion image. Then, theinput/output control unit 260 displays the above-described input screenof the detection failure information. Next, in step S161, theinput/output control unit 260 receives an input of the detection failureinformation. Then, in step S162, the feedback data generation unit 275generates feedback data based on the inputs received in steps S160 andS161.

The fifth example embodiment has been described above. According to thepresent embodiment, the feedback data about the failure of detection bythe lesion detection unit 220 is generated. Therefore, it is possible toobtain data useful for improving the accuracy of the detectionprocessing of the lesion detection unit 220.

Various modifications can be considered for each of the above-describedexample embodiments. For example, in the above-described exampleembodiments, the output image acquisition unit 210 acquires the capturedimage from the processor device 130 in real time, and the lesiondetection unit 220 performs the detection processing of the lesion inreal time, but these processing may not be performed in real time. Inthe above-described example embodiments, each of the informationprocessing devices 200, 201, 202, and 203 includes the lesion detectionunit 220, but since each of the information processing devices 200, 201,202, and 203 only needs to be able to acquire the log of the detectionprocessing, the lesion detection unit 220 may be provided in anotherdevice. In this case, the detection log acquisition unit 240 acquiresthe log of the detection processing from another device. Further, thefeatures of the above-described second to fifth example embodiments maybe arbitrarily combined. In the above-described example embodiments, theexamination support system 10 includes the display device 140 and thedisplay device 300, but all the displays may be performed on any one ofthe display devices.

Although the present invention is described above with reference to theexample embodiments, the present invention is not limited to theabove-described example embodiments. Various modifications that can beunderstood by those skilled in the art can be made to the configurationand details of the present invention within the scope of the presentinvention.

Some or all of the above-described example embodiments may also bedescribed as in the following supplementary notes, but are not limitedto the following.

(Supplementary Note 1)

An information processing device comprising:

a first acquisition unit configured to acquire a capturing time of alesion image instructed to be saved by a user, from a series of imagescaptured by an endoscope during examination with the endoscope;

a second acquisition unit configured to acquire a capturing time of alesion image detected by detection processing for the series of imagescaptured by the endoscope during the examination; and

a display control unit configured to cause a display device to display afirst capturing time and a second capturing time which are plotted on atime axis, the first capturing time being the capturing time acquired bythe first acquisition unit, the second capturing time being thecapturing time acquired by the second acquisition unit.

(Supplementary Note 2)

The information processing device according to Supplementary note 1,wherein

the first acquisition unit further acquires a first lesion image that isthe lesion image instructed to be saved by the user,

the second acquisition unit further acquires a second lesion image thatis the lesion image detected by the detection processing, and

the display control unit displays the first lesion image in associationwith the first capturing time, and displays the second lesion image inassociation with the second capturing time.

(Supplementary Note 3)

The information processing device according to Supplementary note 2,further comprising an image selection unit configured to specify aplurality of the second lesion images in which the same lesion site isdetected and to select a representative image from the plurality of thesecond lesion images,

wherein the display control unit displays the second lesion imageselected as the representative image from the plurality of the secondlesion images in which the same lesion site is detected.

(Supplementary Note 4)

The information processing device according to Supplementary note 2 or3, further comprising a feedback data generation unit configured to,based on an instruction to select the first lesion image or the secondlesion image input from the user and detection failure information ofthe selected lesion image input from the user, generate feedback data inwhich the lesion image and the detection failure information areassociated with each other,

wherein the detection failure information is information indicating, asa type of detection failure, either of erroneous detection of a lesionand detection omission of a lesion.

(Supplementary Note 5)

The information processing device according to any one of Supplementarynotes 1 to 4, further comprising a predetermined site image specifyingunit configured to specify, from the series of images, an image in whicha predetermined site of a body cavity is captured,

wherein the display control unit plots the second capturing time after acapturing time of the image in which the predetermined site is captured,on the time axis.

(Supplementary Note 6)

The information processing device according to Supplementary note 2,further comprising an output image acquisition unit configured toacquire the series of images output from an image output deviceconfigured to output the image captured by the endoscope, wherein

the image output device is a device configured to output an image havinga predetermined feature until a certain time elapses from an instructiontime when the user instructs to save a lesion image during theexamination with the endoscope,

the first acquisition unit acquires the capturing time of the lesionimage instructed to be saved by the user and the lesion image bydetermining a time when an instruction to save is generated based on thepresence or absence of the feature in the image acquired by the outputimage acquisition unit.

(Supplementary Note 7)

The information processing device according to any one of Supplementarynotes 1 to 6, wherein the display control unit displays a scale of thetime axis in an enlarged or reduced manner, based on the instructioninput from the user.

(Supplementary Note 8)

The information processing device according to any one of Supplementarynotes 1 to 7, wherein

the first acquisition unit further acquires a first lesion image whichis the lesion image instructed to be saved by the user,

the second acquisition unit further acquires a second lesion image whichis the lesion image detected by the detection processing, and

the information processing device further includes a failure detectionunit configured to detect a failure of detection of a lesion site by theuser or the detection processing by comparing the first lesion imagewith the second lesion image.

(Supplementary Note 9)

The information processing device according to any one of Supplementarynotes 1 to 7, further comprising a failure detection unit configured todetect a failure of detection of a lesion site by the user or thedetection processing by comparing the first capturing time with thesecond capturing time.

(Supplementary note 10)

A display method comprising:

acquiring a capturing time of a lesion image instructed to be saved by auser, from a series of images captured by an endoscope duringexamination with the endoscope;

acquiring a capturing time of a lesion image detected by detectionprocessing for the series of images captured by the endoscope during theexamination; and

causing a display device to display a first capturing time and a secondcapturing time which are plotted on a time axis, the first capturingtime being the capturing time of the lesion image instructed to be savedby the user, the second capturing time being the capturing time of thelesion image detected by the detection processing.

(Supplementary note 11)

A non-transitory computer-readable medium storing a program that causesa computer to execute:

a first acquisition step of acquiring a capturing time of a lesion imageinstructed to be saved by a user, from a series of images captured by anendoscope during examination with the endoscope;

a second acquisition step of acquiring a capturing time of a lesionimage detected by detection processing for the series of images capturedby the endoscope during the examination; and

a display control step of causing a display device to display a firstcapturing time and a second capturing time which are plotted on a timeaxis, the first capturing time being the capturing time acquired in thefirst acquisition step, the second capturing time being the capturingtime acquired in the second acquisition step.

REFERENCE SIGNS LIST

-   -   1 INFORMATION PROCESSING DEVICE    -   2 FIRST ACQUISITION UNIT    -   3 SECOND ACQUISITION UNIT    -   4 DISPLAY CONTROL UNIT    -   10 EXAMINATION SUPPORT SYSTEM    -   62 TIME AXIS    -   63 MARK    -   64 MARK    -   100 ENDOSCOPE SYSTEM    -   110 ENDOSCOPE    -   111 INSERTION PORTION    -   112 OPERATION PORTION    -   113 IMAGE CAPTURING PORTION    -   120 LIGHT SOURCE DEVICE    -   130 PROCESSOR DEVICE    -   140 DISPLAY DEVICE    -   200 INFORMATION PROCESSING DEVICE    -   201 INFORMATION PROCESSING DEVICE    -   202 INFORMATION PROCESSING DEVICE    -   203 INFORMATION PROCESSING DEVICE    -   210 OUTPUT IMAGE ACQUISITION UNIT    -   220 LESION DETECTION UNIT    -   230 OPERATION LOG ACQUISITION UNIT    -   240 DETECTION LOG ACQUISITION UNIT    -   250 IMAGE SELECTION UNIT    -   260 INPUT/OUTPUT CONTROL UNIT    -   265 PREDETERMINED SITE IMAGE SPECIFYING UNIT    -   270 FAILURE DETECTION UNIT    -   275 FEEDBACK DATA GENERATION UNIT    -   300 DISPLAY DEVICE

What is claimed is:
 1. An information processing device comprising: atleast one first memory storing program instructions; and at least onefirst processor configured to execute the instructions stored in thefirst memory to: acquire a capturing time of a lesion image instructedto be saved by a user, from a series of images captured by an endoscopeduring examination with the endoscope; acquire a capturing time of alesion image detected by detection processing for the series of imagescaptured by the endoscope during the examination; and cause a displaydevice to display a first capturing time and a second capturing timewhich are plotted on a time axis, the first capturing time being thecapturing time of the lesion image instructed to be saved by the user,the second capturing time being the capturing time of the lesion imagedetected by the detection processing.
 2. The information processingdevice according to claim 1, wherein the first processor is furtherconfigured to execute the instructions to: acquire a first lesion imagethat is the lesion image instructed to be saved by the user, acquire asecond lesion image that is the lesion image detected by the detectionprocessing, and perform control so as to display the first lesion imagein association with the first capturing time, and display the secondlesion image in association with the second capturing time.
 3. Theinformation processing device according to claim 2, wherein the firstprocessor is further configured to execute the instructions to: specifya plurality of the second lesion images in which the same lesion site isdetected, select a representative image from the plurality of the secondlesion images, and perform control so as to display the second lesionimage selected as the representative image from the plurality of thesecond lesion images in which the same lesion site is detected.
 4. Theinformation processing device according to claim 2, wherein the firstprocessor is further configured to execute the instructions to, based onan instruction to select the first lesion image or the second lesionimage input from the user and detection failure information of theselected lesion image input from the user, generate feedback data inwhich the lesion image and the detection failure information areassociated with each other, and the detection failure information isinformation indicating, as a type of detection failure, either oferroneous detection of a lesion and detection omission of a lesion. 5.The information processing device according to claim 1, wherein thefirst processor is further configured to execute the instructions to:specify, from the series of images, an image in which a predeterminedsite of a body cavity is captured, and perform control so as to plot thesecond capturing time after a capturing time of the image in which thepredetermined site is captured, on the time axis.
 6. The informationprocessing device according to claim 2, wherein the first processor isfurther configured to execute the instructions to acquire the series ofimages output from an image output device configured to output the imagecaptured by the endoscope, the image output device comprise at least onesecond memory storing program instructions and at least one secondprocessor configured to execute the instructions stored in the secondmemory to output an image having a predetermined feature until a certaintime elapses from an instruction time when the user instructs to save alesion image during the examination with the endoscope, and the firstprocessor is further configured to execute the instructions to acquirethe capturing time of the lesion image instructed to be saved by theuser and the lesion image by determining a time when an instruction tosave is generated based on the presence or absence of the feature in theimage acquired from the image output device.
 7. The informationprocessing device according to claim 1, wherein the first processor isfurther configured to execute the instructions to perform control so asto display a scale of the time axis in an enlarged or reduced manner,based on the instruction input from the user.
 8. The informationprocessing device according to claim 1, wherein the first processor isfurther configured to execute the instructions to: acquire a firstlesion image which is the lesion image instructed to be saved by theuser, acquire a second lesion image which is the lesion image detectedby the detection processing, and detect a failure of detection of alesion site by the user or the detection processing by comparing thefirst lesion image with the second lesion image.
 9. The informationprocessing device according to claim 1, wherein the first processor isfurther configured to execute the instructions to detect a failure ofdetection of a lesion site by the user or the detection processing bycomparing the first capturing time with the second capturing time.
 10. Adisplay method comprising: acquiring a capturing time of a lesion imageinstructed to be saved by a user, from a series of images captured by anendoscope during examination with the endoscope; acquiring a capturingtime of a lesion image detected by detection processing for the seriesof images captured by the endoscope during the examination; and causinga display device to display a first capturing time and a secondcapturing time which are plotted on a time axis, the first capturingtime being the capturing time of the lesion image instructed to be savedby the user, the second capturing time being the capturing time of thelesion image detected by the detection processing.
 11. A non-transitorycomputer-readable medium storing a program that causes a computer toexecute: a first acquisition step of acquiring a capturing time of alesion image instructed to be saved by a user, from a series of imagescaptured by an endoscope during examination with the endoscope; a secondacquisition step of acquiring a capturing time of a lesion imagedetected by detection processing for the series of images captured bythe endoscope during the examination; and a display control step ofcausing a display device to display a first capturing time and a secondcapturing time which are plotted on a time axis, the first capturingtime being the capturing time acquired in the first acquisition step,the second capturing time being the capturing time acquired in thesecond acquisition step.